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This docusent reports the results of an evaluation of 
the protocol fils series "Concepts and Patterns in Teacher Pupil 
Interaction.** Both concept acquisition and user reactions were 
evaluated. Concept acquisition was tested using two studies, a pre* 
and posttest design and a posttest onlv design. Experisental groups 
in both studies scored significantly higher on the posttest than the 
control group did, and the pre* and posttest group scored higher 
gains than the posttest only group did. To test user miction, an 
instructor questionnaire consisting of 1U Likert scale 1 dss and a 
student questionnaire consisting of 20 itess were developed. 
Instructors felt that: (a) the quality of the fill was good; (b) the 
concepts stressed in the series were clearly defined and 
understandable; and (c) the saterials were applicable in a vide 
variety of instructional sethods, groups, and settings. Analyses of 
student responses show that they felt that: (a) the filss were worth 
the class tise spent on thee, (b) the purpose of the filss was clear, 
and (c) the concepts esphasized in the fill would be valuable to thes 
in the classroos. Results of the study show that use of the fill led 
to an overall gain in the ability of students to categorize classroos 
behavior in terss of a specified set of concepts. The conceptual 
clarity, instructional utility, and technical quality of the filss in 
the series were established for a heterogeneous sasple of graduate 
and undergraduate pre service and inservice teachers. (RNB) 
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Foreword 

BEST COPY AVAILABLE 
In 1970-71, a pilot protocol film based on concepts In teacher- 
pupil Interaction was developed and produced by the Indiana University 
Protocol Materials Project. This film, entitled Classroom Protocols , 
was used and evaluated with necessarily rudimentary Instruments and 
techniques and the results reported as a pilot evaluation studyJ 

Drawing upon the conceptual and technical experiences Ittvc^lved In 
this preliminary project, a second set of protocol films was produced 
In 1971 as part of a complementary set of protocol and training 
materials. These materials were developed as an exploratory product 
of the National Center for the Development of Training Materials In 
Teacher Education, at Indiana University. A better designed set of 
films, entitled Patterns In Teacher-Pupil Interaction : Reacting to 
Pupil Responses I & II. and a more soundly based set of evaluation 
instruments made possible a more systematic evaluation, one that has 
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been formally reported. 

Finally, during the years 1971-73, the protocol films In the 
Concepts and Patterns in Teacher-Pupl 1 Interaction series were developed , 
and produced, as an extension and culmination of these Initial projects. 
The present report describes a systematic evaluation of the films In 
the Concepts and Patterns series. It should be of Interest, first, 
to users and potential users of these protocol films since It does 
summarize the results of using the Concepts and Patterns series In 
classroom Instruction over a wide variety of settings, levels, and 
students. The results reported also have certain Implications for the 
general design of protocol materials; for that reason, they should be 
of Interest to developers of protocol materials. Finally, users and 
developers alike may well find something of value In the practical 
solutions to the evaluation problems described In this report. 



"A Preliminary Evaluation Report on the Development and Use of Filmed 
Protocol Materials Within Two Instructional Strateglesi" Unpublished 
report. Bloomlngton, Indiana: Indiana University Protocol Materials 
Project. July. 1971. 

'Kliucker, Joy. "Effects of Protocol and Training Materials." Acquiring 
Teaching Competencies: Reports and Studie s, Bloomlngton, Indiana: 
National Center for the Development of Training Materials In Teacher 
Education. Report #6. September, 1974. 
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Effec ts of a Protocol F1 Tjn lejr iM. ifl Igmi BEST COPY AVAILAB6 
of learning Outcowies and Reactions of Users 

The primary purpose of this report Is to summarize the results 
of a systematic evaluation of a specific set of protocol materials. 
However, since the general development and evaluation of protocol 
materials Is In a formative staqe, the content of this report may 
well have more general Implications. These Implications pertain 
especially to the design of protocol materials themselves and of the 
strategies and Instruments for their evaluation. 

The specific protocol materials evaluated In this report comprise 
the films In the Concepts and Patterns In Teacher-Pupil Interaction 
series. The purpose of these protocol films Is to specify, define and 
Illustrate a set of concepts pertaining to teacher behavior In class- 
room Interaction with pupils. When used appropriately, these films 
should contribute directly to the acquisition of a set of Interpretive 
concepts and Indirectly to the development of the skills described by 
these concepts. In terms of learning outcomes, however, the present 
report focuses on the acquisition of concepts, the primary outcome at 
which the use of any protocol material is directed.^ 

Description of the Protocol Series 

The Concepts and Patterns series is based on three pairs of concepts: 
approving and disapproving, probing and inrorming, reproductive question- 
ing and productive questioning. As indicated in the previous paragraph, 
these concepts refer to teacher behavior in an interactive model of 
teaching. In other words, they refer to teacher solicitation and 
reaction in teacher- led classroom discussion. Definitions of the concepts 
are provided in the Appendix to this report. 



^Gliessman, David. "An Introduction to Protocol and Training Materials." 

Acquiring Teaching Competencies: Reports and Studies . Bloominqton, 

Indiana: National Center for the Development of Training Materials in 

Teacher Education. Report #1, September, 1972. 
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The series Itself consists of a set of nine motion picture films 
In color with sound. These films portray or document teacher-pupil 
Interaction In varied subject areas and at varied grade levels. The 
nine films are divided Into tMO subsets: 

a. Three Concept films that define. Instance, and explicate 
the basic concepts; these films are also designed to 
provide a motivational and Informational "set" for the 
remaining films In the series. These are narrated films 
ranging from five minutes to ten minutes In length. 
The titles and a brief description of each follows: 

Approving and D1 sapproving— 1 n troducwS , defines, analyzes, 
ana Insiancis "approval" and "disapproval" as teacher 
reactive behaviors. 

Probing and Informing— Introduces , defines, and Instances 
"problng^nd "Informing" as teacher reaction skills In 
classroom discussion. Also analyzes the concept of 
"probing" as a questioning technique. 

Questioning: Reproductive and Productlve- lntroduces, 
defines, and Instances "reproaiicnve questioning" and 
"productive questioning" as teacher questioning skills. 
Analyzes "productive questioning" as Incorporating higher 
level goals In Bloom's Taxonomy. Compares "reproductive 
questioning" (In other words, questioning that requires 
essentially memory level responses) with the "knowledge 
level" of the taxonomy, 
b. Six Pattern films each documenting an episode In teacher- 
pupil Interaction. Different films In this subset are 
Interpretable In terms of the set cf concepts described 
above since the relevant teacher behaviors occur with 
significant frequency. However, the salience of the 
behaviors Is not artificially Increased by editorial 
devices: the films are not narrated and editorial 
structuring devices (such as title frames within the film) 
are not used. These may be described as essentially 
documentary films. The films In this subset are approxl- 
metely ten minutes In length. Titles and a brief 
description of each episode follows: 
People Who Help Us— A second grade teacher and her class 
expl ore"^e"foplFof "coimiunlty helpers". 
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The Five Senses— Also at the second grade level, this 
teacher explores the concept of the five senses with 
her class. 

Ecology— During a field trip, a seventh grade teacher 
analyzes the problem of environmental pollution with 
her students. 

Essentials for life— A seventh grade class recitation on 
the basic essentials for life followed by a discussion 
using this Information In exploring an original problem. 

Population Control— A tenth grade social studies class 
discusses the problem of population growth and population 
control. 

Bacteria— A tenth grade science class reviews the forms 
ana uses of bacteria, then applies this Information to 
a demonstration of bacterial action. 

A distinctive characteristic of this protocol series Is that It 

Is entirely "film based"; no adjunct material Is provided for student 

use. The practical consequence of this ;s that the film series Itself 

must convey essential conceptual content as well as portray behavior 

to be Interpreted.^ The purpose of such a format was to provide an 

Instructional resource that was highly adaptable to the varied 

approaches of Instructors In normal classroom use. In oth<*r words, 

the film-based format was designed to avoid the constraints on 

Instruction often Imposed by a "multi-media package". To further 

assure such adaptability, the films within the series are flexibly 

sequenced. Although the rationale of the series does suggest using 

specific Concept films and Pattern films in that sequence, the selection 

of individual films is largely open to the instructor and variations 

in sequence are encouraged. 

General Evaluation Plan 

Although the details of instrument development and data gathering 
are described in the following sections of this report, it should be 



A ten page manual does provide the instructor with definitions of the 
the concepts, suggested uses of the films, selected references, etc. 
The instructor may certainly draw upon this resource to elaborate 
upon the concepts. However, even in this brief manual, appropriate 
textual material that might be reprinted for students amounts to no 
more than a few pages. 
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helpful at this point to provide a general sumnary of, and rationale 
for* the evaluation plan. Both learning outcomes (In terms of concept 
acquisition) and reactions of users were of Interest to the developers 
of the series. A decision was made to collect data on these two effects 
of the series largely In different settings. In fact. In only two of 
the 23 classes eventually Included In the study were data obtained on 
both learning outcomes and reactions of users. 

In the case of assessing learning outcomes, data were gathered 
In those settings in which the Instructional procedure and conditions 
of testing could be appropriately controlled. To fall to exercise such 
controls would certainly lead to unlnterpretable, perhaps specious* 
results. For this reason, all data on concept acquisition were collected 
In classes with which project personnel were In regular and direct con- 
tact, either as Instructors or as evaluators. 

This degree of control* on the other hand, was not a desired 
condition for assessing the reactions of users. In this case, a 
deliberate attempt was made to assure Independence from project staff 
control (or even staff contact). In other words, the survey of users 
was designed to be as analogous as possible to the ultimate conditions 
of use during general distribution of the film series. For this reason. 
Instructors Interested In participating In the survey were essentially 
treated as consumers of the films. They were provided the films In 
the series. Instructor's manual, and necessary evaluation materials 
without added support. Instructions or orientation. In the majority 
of cases* there was no personal contact during Instruction between the 
project staff and the Instructors. It was felt that this arrangement 
would yield the best Information on the completeness and applicability 
of the series In "normal" use. 
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Learning Outcomes as Evidenced by Concept Acquisition 

The learning criterion In this study was the acquisition of the 
six concepts named on the first page of this report. For purposes of 
this study, concept acquisition was defined as the ability to Identify 
and categorize correctly recorded Instances of teacher behavior 
specified by this set of concepts. This criterion Is consistent with 
the general purpose of protocol materials: to aid In the functional 
acquisition of concepts as a basis for Interpretive skills. 

At an early point In development. It was decided to construct an 
evaluation Instrument that would require the learner to Identify and 
categorize audio-visual records of behavior without supporting printed 
records or (descriptions. It was felt that categorizing such observed 
behavior would most closely approximate the Interpretation of behavior 
in actual classrooms.^ This decision called for the development of a 
test format In which classroom episodes would be presented by means of 
videotape or film. In the following two sections, this evaluation 
Instrument and the general design used to assess Its validity will be 
described. 

Evaluation Instrument 

The basic evaluation Instrument, Categorizing Teacher Behavior. 
consists of 30 brief classroom episodes presented by means of videotape 
transferred from film. Each episode Is presented twice with a delay 
of from two to four seconds between presentations; the purpose of this 
repeated presentation Is to minimize dependence on memory as a factor 
In responding. Following the second presentation of each episode, the 
student Is allowed a fifteen second period In which to respond by 
recording his answer on a separate answer sheet. 



^Ultimately, a printed evaluation Instrument was developed following 
the same format as the Instruments described In this section but 
containing different episodes. This "paper and pencil" form was 
developed partly as an exploratory Instrument and partly as an 
efficient test form for users of the series. 
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Two forms of the test were developed. Form F1 and Form F2. The 
only difference between the two forms Is In the source of the classroom 
episodes portrayed. In the case of Form F1, the classroom episodes 
were taken from "outtakes" of film footage from the Concepts and 
Patterns series (that Is, from footage not Included In the films 
themselves). The practical consequence of this Is that the classroom 
setting, general subject matter, teachers and pupils are the same In 
the test videotape as In the film series Itself. However, the specific 
episodes and behaviors appearing In the videotape test In no case appear 
In the actual films of the series. To compensate for some of the 
obvious methodological problems associated with evaluating performance 
using the same source of materials as that used In Instruction, an 
alternate form of the test was also constructed. This form. Form F2, 
Incorporates episodes from the pilot protocol films produced by the 
Indiana University Project; In this case, none of the classroom settings, 
teachers, pupils or episodes are comnon to both Instructional and 
evaluation conditions. In other words, no footage from the production 
of the final film series was used In the development of Form F2. 

The task for the student in responding to either Form F1 or 
Form F2 is the same: determining whether or not the teacher's 
behavior in a given episode Instances or exemplifies each of a 
selected four of the six concepts Included in the series. As an 
example, in the case of one episode, the student is asked to indicate 
whether or not he sees evidence of probing, informing, approval and 
disapproval in the teacher's behavior. Each form of the test is 
divided into three major sections, providing for all possible combin- 
ations of the pairs of concepts. In the case of each form. Part I 
requires episodes to be categorized in terms of probing, informing, 
approval and disapproval. In Part II, the concepts involved are 
productive questioning, reproductive questioning, approval and dis- 
approval. In Part III, the concepts are probing, informing, productive 
questioning and reproductive questioning. The total time for the test 
( including repeated episodes, delays between episodes, and delays for 
responding) is approximately 35 minutes (with some small variation in 
time between forms). 
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In the present study, student responses were scored as correct 
(that Is, an accurate cateqorlzatlon of the teacher's behavior); 
partially correct/partially incorrect (that Is, an Incomplete cate- 
qorlzatlon of the teacher's behavior); or Incorrect (an Inaccurate 
cateqorlzatlon of the teacher's behavior). Two points were allowed 
for each correct response and one point was allowed for each partially 
correct/partially incorrect response. With four concepts to be cate- 
gorized for each episode, and thirty episodes in all, a perfect score 
on the test is 240 points.^ 

Design for Instrument Validation 

The desiqn for establishinq the validity of the evaluation 
instrument described in the previous section was primarily one of 
construct validation. Primary emphasis was placed upon planning 
specifically for the content validity of the Instrument, collection 
of evidence on consistency of scores, and collection of evidence on 
the effect of specified treatments on scores. To contribute to con- 
tent validity, each form of the test was constructed with an equal 
number of items for each of the six concepts (20 items for each con- 
cept). Consistency of scores was assessed through a comparison of 
performance on matched halves of the test. The effect of treatment 
on scores is reflected in the data reported under the "Results" sec- 
tion of this report. Detailed summaries of evidence on the validity 
of the evaluation Instrument, as well as a rationale for the method 
of validation, will be reported in a test manual currently under pre- 
paration. For purposes of the present report, it is sufficient to 
say that the evidence for validity is very adequate for the purposes 
of the Instrument. To put it succinctly, the test. Categorizing 
Teacher Behavior , was consistently sensitive to instructional treat- 
ment utilizing the protocol films in the series. 



After completion of the studies in this report, the allotment of 
partial credit in scoring was discontinued, retaining a simple 
"correct" and "Incorrect" scorinq system. The point total of 240, 
however, was retained to represent a perfect score. 
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Instructional Procedure 

The protocol films In the Concepts a;id Patterns series were used 
as part of regular classroom Instruction In five sections of a master's 
level course In educational psychology. One section of the course was 
taught by one of the authors, while the remaining four sections were 
taught by a faculty member attached to the protocol materials project 
staff. 

The students enrolled In these sections were quite representative 
of those normally enrolled In the course. They were a heterogeneous 
group largely seeking masters degrees In teaching or professional 
education but having diverse professional goals (e.g. elementary, 
secondary and college teaching; educational administration; media 
production and administration; guidance and counseling). Both pre- 
servlce and practicing teachers were enrolled. Both males and females 
were enrolled In approximately equal numbers and represented a broad 
age range. 

The topics of teacher behavior and teacher-pupil Interaction are 
a conventional part of this educational psychology course as It Is 
normally taught by the Instructors Involved In this study. Students 
enrolled In these sections of the course v.'ould see those topics as 
appropriate to the course content. Both Instructors also traditionally 
make use of fUm of various kinds as part of their classroom Instruc- 
tion. In the present study, the Instructors used six or more of the 
films In the Concepts and Patterns series. Incorporating these films 
Into general classroom lecture and discussion. The Instructional pro- 
cedure Involved some variation on a standard sequence: showing a single 
Concept film followed by one or more Pattern films. Discussion and 
analysis of the concepts and of the filmed behavior both preceded and 
followed film showings. 

Within these sections, however, there were certain differences In 
course procedure, evaluation design, and test form used. For thl;* 
reason, the results will be treated under two separate studies. Differ- 
ences In evaluation design will be clarified In the following section; 
differences In course procedure and test form can be sunmarlzed briefly: 

Study I. Four sections of the course with a total enrollment 
oTSS* students were Instructed during the regular academic 
year, 1973-74. In these sections, film-based instruction 
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covered from six to eight hours of classroom Instruction 
over a two three week period. In the case of these 89 
students. Form F1 of the videotape test was used as the 
primary evaluation Instrument. 

Study II. During a "short course" offered during the 
summer session, 1974, nine students were given more 
Intensive film-based Instruction. With this group, 
approximately 12 hours of Instruction was condensed 
into four calendar days. In addition to the films In 
the series. Form F1 of the videotape test was used 
Instruct onally as material for analysis. Form F2 
was used as the evaluation Instrument. An additional 
section of the same course (taught by an Instructor 
who was not a member of the project staff and who did 
not generally use Instructional films) was used to 
obtain comparison data on Form F2. A total of 35 
students were enrolled In this section. No training 
with the rrotocol films In the series was Incorporated 
Into this section. 



Results 

In Study I, the 89 students enrolled In the three sections were 
randomly divided, within sections. Into two groups: a "posttest only" 
group that received film-based training followed by a single adminis- 
tration of Form Fl of the test; and a pre- and posttest group that 
received Form Fl as a pretest, received film-based training, and repeated 
the same test form after training.^ The purpose of this differential 
testing treatment was to help ascertain (and thus allow for) any practice 
or learning effect from pretesting. It should be noted that students In 
both Study I and Study II who received the videotape test as a pretest 
were given a one page summary of the concept names and definitions to 
"study" for approximately five minutes before taking the pretest. This 
procedure. Intended to Increase the "rationality" of taking the videotape 
test, almost certainly provided minimal training that Increased scores 
on the pretest. It seemed to the authors, however, to be the only 
defensible way to avoid exaggerating gains on post testing. 



W to a slight error In the randomization procedure, seven more 
students were Included In the Post Only Group than In the Pre-Post 
Group. 
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The results of Study I are summarized In Table 1 below. 

Table 1. Results of Protocol Film Based Training for Pre-Post and Post 
Only Groups . 



Pretest Posttest 
N y SD y SD 



41 193.34 20.17 212.71 10.13 

Post Only „ 

Group — 48 203.00 12.31 



Difference 
Tested : 

Pretest vs. Posttest 
for Pre«Post Group . 

Pre-Pos t Group vs. 
Post OfTfy'TSrou'p on 
posttest. 

Pretest for Pre-Post 
Group vs. posttest for 
p^FOnly Group . 



63.33 1. 40 <.01 
15.78 1. 87 <.01 
7.49 1. 87 <.01 



The Pre-Post Group showed a gain in mean test performance of 19.37 points, 
from 193.34 to 212.71. This gain resulted in an F(1, 40) • 63.33 which 
was significant beyond the .01 level. To check on a possible practice 
or learning effect from taking Form F1 as a pretest, the mean test 
performance of the Post Only Group was compared to the after training 
performance of the Pre-Post Group. The Post Only Group mean of 203.00 
was significantly less than the mean test performance of the Pre-Post 
Group F(1, 87) • 15.78, (p <.01). A comparison of the mean test per- 
formance of the Post Only Group (203.00) with the before training test 
performance of the Pre-Post Group (193.34). however, yielded an 
F(1, 87) • 7.49 which was significant beyond the .01 level. 

These results suggest that the gain in test performance of the 

1. 1— 
ERIC 
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Pre-Post Group reflected partly a gain In performance from training and 
partly the contribution of the pretesting experience on Form Fl either 
through a practice effect or through an Interactive effect with training. 
There clearly Is, however, a discernable and significant gain In mean 
test performance attributable to training alone. 

The results for Study 11 are summarized In Table 2 below. 



Table 2. Results of Intensive Protocol film Based Training Compared 
to No Training. 





Pretest 




Posttest 


N 


J SD 




7 SO 


Untrained 

Group 35 


189.00 13.95 




190.62 15.86 


Protocol 

Group 9 


100 11 14 A7 




9nfl 7ft R 11 


Difference 
Tested 

Untrained Group 
vs. Protocol Group 
on Pretest 


F 

0.03 


df 

1. 42 


>.25 


Untrained Group 
Pretest vs. Posttest 


0.76 


1, 34 


>.25 


Protocol Group 
Pretest vs. 
Posttest 


28.42 


1. 8 


<.01 


Untrained Group vs. 
Protocol Group on 
Posttest 


9.17 


1, 42 


<.01 



In this case, both the Untrained Group and Protocol Group were 
pretested and posttested with Form F2 of the videotape test. On the 
pretest, the difference In mean test performance between the two groups 
was only 0.89, a difference of less than one Item score. This difference 
yielded an F(l, 42) • 0.03 which failed to reach significance (p >.25). 
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Thus, the Initial performance of the Untrained Group and Protocol Group 
can be considered equivalence. 

The pre- and posttest results for the Untrained Group show a qain 
of 1.62 points In mean test performance (from 189.00 to 190.62). An 
F(1, 34) • 0.76 for this difference failed to reach significance 
(p > .25). It Is Interesting to note. In this case, that the practice 
effect of repeating the same form of the videotape test was negligible, 
fhls suggests that some of the gain shown by the Pre- Post Group In 
Study I was a function of an Interaction of learning on the pretest 
with training Itself rather than a simple practice effect. 

The results for the Protocol Group show a gain of 20.67 points 
(from 188.11 to 208.78). A test for the significance of this gain 
yielded an F(1, 8) « 28.42 which Is significant beyond the .01 level. 
In essence, then, training with the use of a protocol film based 
strategy and with intensive analysis of Form F1 of the videotape as 
an instructional medium resulted in a highly significant gain in test 
performance. The gain was attributable neither to a simple practice 
effect from repeating the videotape test nor to the particular selection 
of students. 
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Characteristics of Use and Reactions of Users 

The primary qoal of the survey of users was to arrive at an 
objective estimate of the Instructional value and applicability of the 
films In the Concepts and Patterns series. It was expected also that 
the data obtained would provide an Indirect estimate of the neneral 
applicability of protocol materials deslqned similarly to the films 
In this series. As Indicated In the Introduction to this report, the 
Concepts and Patterns series was deslqned to meet two practical criteria 
In particular: (1) flexibility for the Instructor In selecting and 
sequencing films for showing; (2) treatment of the necessary conceptual 
content within the film series Itself rather than In adjunct materials.^ 
Evidence on the effect of these design characteristics mlqht well have 
Implications for the future development of protocol materials. 

To answer these and related questions, two general kinds of data 
were gathered: (1) descriptive data on conditions and characteristics 
of use and (2) reactions of Instructors and students to the conceptual 
clarity. Instructional value, and technical quality of the films. 

Survey Sample 

During the academic year. 1973-74. 23 Instructors were qlven the 
opportunity to use. free of cost, the Concepts and Patterns film series. 
Of the 23 who were contacted. 18 requested use of the films. These 
18 Instructors returned both Instructor and student evaluation forms. 
A total of 400 students from 20 classes reacted to the use of the films 
during this survey. 



the term "necessary conceptual content" Is used advisedly to Indicate 
treatment of conceptual content necessary to the analysis of the films 
(e.g. definitions, some degree of analysis, suggesting relationships 
to other concepts). The term In no way should Imply a full explication 
or elaboration of conceptual content within a theoretical system. 
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The settings In which the films were used were representative of 
the types of settlnns for which they were developed. For an under- 
graduate setting, 10 undergraduate classes were used which ranged from 
Introductory courses In educational psychology to preservlce methods 
courses. All but two were located on the Bloomlngton campus of 
Indiana University. The two exceptions were a university In the eastern 
United States and a teacher training college In West Germany. For a 
graduate setting, six graduate education classes were used. These 
classes consisted of a general teacher education course, four 
educational psychology courses, and a course In educational adminis- 
tration. They were all located on the Bloomlngton campus of Indiana 
University. Finally, for an Inservlce setting, four training groups 
were used In two different locations In the midwest. 

Instruments and Methods 

An Instructor Questionnair e was developed (1) to collect Information 
about the educational setting In which the films were used, (2) to obtain 
Information regarding the specific use of the films (e.g., sequencing 
and Instructional time) and (3) to obtain Instructor reactions on 14 
Likert Items to the conceptual. Instructional and technical quality of 
the films. The Instructor Questionnaire was distributed to the In- 
structors at the time they received the films. The completed forms were 
returned along with the films and were Immediately checked for complete- 
ness of response. A follow up was conducted. If necessary, to obtain 
responses to all Items on the form. In all cases, the Instructors were 
cooperative and willing to take time to complete the evaluation form. 
Therefore, It was Judged that the Information obtained was accurate and 
reliable for the stated objectives of the evaluation. 

Similarly, a Student Rating Scale was developed to obtain student 
reactions to the conceptual. Instructional and technical quality of the 
films. The Student Rating Scale was administered by the instructors 
who participated In the survey. Anonymity of student response was 
allowed In order to obtain more candid student reactions. As a result, 
no Individual student background data were obtained. Consequently, the 
student reaction data could not be related to other student variables 
such as teaching experience or educational background. 



BEST COPY AVAILABLE 



Validation of the Student Rating Scale , During the development 
of this instrument* a panel of judges assessed possible scale items 
for clarity and content relevance. As a result, 24 Likert items were 
selected for inclusion in the initial version .of the Student Rating 
Scale . Since student responses were necessary for establishing the 
scale characteristics of a final version of the instrument, five 
classes (n » 106) of the total sample were isolated for validation 
data. Of these five classes, one was in an inservice setting, one in 
an undergraduate setting, and three in a graduate setting. The five 
classes, taught by four instructors, consisted of 14, 18, 22, 24 and 
28 students. 

For each of the 24 Likert items on the Student Rating Scale , 
item-total correlation coefficients were calculated within each of 
the five classes. The positive differences between these item- total 
correlation coefficients and zero were tested using the .05 level of 
significance. If an item failed to reach t|ie .05 level of significance 
in at least two of the five classes, the item was subjected to detailed 
scrutiny by a panel of three judges. Ultimately, this panel reviewed 
seven items using the criteria of clarity, importance, and generality. 
All other items reached significance in at least four of the five 
classes and were judged to have characteristics which would permit 
reliable interpretation. Of the seven items subjected to review, four 
were eventually omitted from the scale because an apparent lack of 
either clarity or generality confirmed the statistical findings. 

As a result of this procedure, a set of 20 items was finally 
identified for evaluating student reactions to the films. The item- 
toval correlation coefficients and scale characteristics (based on 
a pooling of the five classes composing the validation sample) are 
reported in the Appendix in Table A. Each of the twenty item-total 
correlation coefficients were significantly greater than zero 
(df • 104, p < .05). The scale reliability was .89. These scale 
characteristics were judged sufficient to permit evaluation of the 
films. 
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Results of Instructor Evaluation 

The use of a portion of the total survey samole for the validation 
study described In the previous section resulted In a final evaluation 
sample of 15 classes (or "qroups") taught by 14 Instructors, since one 
Instructor was responsible for two classes. Evaluative Information on 
this sample, obtained by means of the Instructor Questionnaire , was 
divided Into two major categories: characteristics of use and 
reactions of Instructors. 

Characteristics of use. As stated earlier, the films were used 
with undergraduate, graduate, and Inservlce groups. Included In these 
groups were preservlce teachers, practicing teachers, and school 
administrators. The 15 groups ranged In size from small work groups 
of as few as six students to a relatively large group of 58 students. 
The average size of the groups was approximately 20 students. 

The method of using the films varied among the Instructors. Eight 
Instructors used the films In lecture or discussion with the total 
class, five Instructors used the films In small groups within a class, 
and one Instructor used both total class situations and small work 
groups. Since the complete film series was available to each Instructor, 
the frequency with which each film In the series was used might Indicate 
the comparative appeal or usefulness of the Individual films. The 
frequency of film selection and the number of times each film was 
shown are reported In Table 3 on the following page. 

Clearly, the Concept films were used by more Instructors than 
were the Pattern films. It appears also that the Concept films were 
shown to the same students more times than the Pattern films. The 
average number of times the Concept films were shown by a single 
Instructor was 1.33 times. For the Pattern films, the average number 
of times shown was from 1.0 to 1.1. It appears that the Concept films, 
designed to present the conceptual content of the series, were 
sufficiently rich In content and clear In structure to warrant multiple 
showings. It was originally speculated th t the Pattern films typically 
would be shown a greater number of times to the same students than 
would the Concept films. This might be especially true If the Pattern 
films were used In work sessions requiring active student participation. 
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Table 3. Frequency of Use and Number of Times Films Were Shown 



Film 


Instructors 


Times Shown 




N 


N 


Average 


Concepts In Teacher- 
Kupll interaction 








proDing ana inTorming 


1 9 


16 


1.33 


Questioning: Reproductive 
and Productive 


12 


16 


1.33 


Approving and Disapproving 


9 


12 


1.33 


Patterns in Teacher- 
Pupil Interaction: 








Bacteria 


7 


8 


1.14 


People Who Help ui 


D 


6 


1.00 


The Five Senses 


7 


7 


1.00 


Population Control 


8 


9 


1.12 


Essentials of Life 


4 


4 


1.00 


Ecology 


S 


5 


1.00 



It was probably the case that very few, if any, of the instructors fully 
exploited the potential that has been built into the Pattern films. 

A review of the Instructor Questionnaire revealed that, of the 14 
Instructors, three showed all the films in the series. More typically, 
two or three of the Concept films and approximately half of the Pattern 
films were used by an Instructor. The sequence in which the films were 
shown differed slightly among the Instructors but by far the most common 
sequence was a Concept film followed by a Pattern film. Twelve 
Instructors used this sequence which was the utilization model on which 
the film series was developed. 

Various amounts of instructional time were devoted to the use of 
the films. Eight instructors used one class period, three Instructors 
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used two class periods, and three Instructors used four class periods. 
Generally* the greater the number of class periods, the greater the 
number of films shown. However, there was some variation among 
Instructors In the relationship of number of class periods to number 
of films shown; one such variation will be reported in the next 
section. 

Reactions of Instructors . The frequency distribution of 
instructors* responses to the 14 items and the item means are reported 
in the Appendix in Table B. Weights were assigned to the adjectival 
responses and item means were calculated. The item means were rounded 
to the nearest whole number and assigned their adjectival meaninqs for 
the following general summary: 

STRONGLY AGREE was the average Instructor response for: 

"The picture quality (color, camera focus, etc.) of the 
films was good." 

"The concepts stressed in the Concept films were clearly 
defined and understandable." 

AGREE was the average Instructor response for: 

"The purpose of the Pattern films was clear." 

"The sound quality of the films was good." 

"The films seemed to hold the attention of my students." 

"The Concept films were at an appropriate level of 
difficulty for my students." 

"Any instructor who wants his students to acquire these 
concepts should use both the Concept films and Pattern 
films." 

"If they were available, I would plan to use these films 
In future courses." 

"The films were obviously produced by professional film 
makers." 

UNDECIDED was not the average response for any item. 
DISAGREE was the average instructor response for: 
"The purpose of the Concept films was unclear." 
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"The classroom episodes portrayed In the films were not 
realistic or believable." 

"Either the Pattern films or the Concept films by 
themselves could have achieved the same purposes." 

"The sequence or format of the Concept films was 
confusing." 

STRONGLY DISAGREE was the average Instructor response for: 

"The concepts stressed In the Concept films were not of 
sufficient Importance to warrant the time expenditure 
Involved." 

Generally, the Instructors' responses were very supportive of the 
film series. The vast ma,ior1ty of the Instructors were very positive 
toward the films on all Items. It will be recalled that the series was 
designed to meet two practical criteria: flexibility In use and 
Inclusion of necessary conceptual content. With respect to these 
criteria, several aspects of the responses are worth noting. 

First, as the previous section on characteristics of use Indicated, 
different Instructors reported using the films In quite varied ways with 
groups of very different size and of different educational or professional 
levels. There was also considerable variation In the number and selection 
of films used by different instructors. This variation In use across 
Instructors, considered In light of their highly favorable reactions In 
general, suggests that the evident flexibility of the film series 
was either reacted to favorably or, at the least, did not mitigate 
against an otherwise favorable reaction. 

With respect to the conceptual content of the series, reactions 
to the Concept films should be particularly noted since this subset of 
the series was deslnned to communicate the necessary conceptual content. 
The numerically smallest Item mean was In response to "The concepts 
stressed In the Concept films were not of sufficient Importance to 
warrant the time expenditure Involved." Since this was a negatively 
stated Item, one can Infer that the typical Instructor thought the 
time spent In using the Concept films was warranted. When this finding 
Is related to the earlier finding that on several occasions the Concept 
films were shown more than once to the same students. It Is fairly well 
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substantiated that the Concept films are "rich" In content. Furthermore, 
the fact that the averaae Instructor response was strongly agree for the 
Item, "The concepts stressed In the Concept fllns were clearly defined 
and understandable", Indicates that the content of the Concept films 
was not only a rich resource but was clear In definition and presentation. 
In general. Instructor reactions to Items concerning the Concept films 
indicates that the conceptual content treated in the films was clear In 
definition, structure and purpose. 

It is quite clear also that the Concept films and Pattern films 
form an Integrated series. As evidence of this, note the average 
Instructor response to these two items: "Either the Pattern films or 
the Ccncept films by themselves could have achieved the same purpose" 
and "Any instructor who wants his students to acquire these concepts 
should use both the Concept films and Pattern films." The Instructors 
disagreed that either the Concept films or the Pattern films could 
have achieved the same objectives by themselves. There was substantial 
instructor support that both the Concept films and Pattern films were 
needed. 

Results of Student Evaluation 

Analyses of the data from the revised Student Rating Scale were 
compiled by using the total scale score as well as the individual item 
scores. The information based on the total scale score is reported 
first, followed by an analysis of class and use characteristics related 
to total scale scores, and finally by a description of the results from 
the individual items. For all of the analyses using the total scale 
score, the class was selected as the unit of analysis. 

General reactions of students . Descriptive information from the 
Student Rating Scale is reported in the Appendix in Table C. For each 
of the 15 classes involved in the survey, the following items are 
included in the table: number of students, total scale mean, standard 
deviation, and reliability. The descriptive information for each class 
is reported in descending order of class means. The total scale score 
has a potential range from a low of 20 to a high of 100. A score of 
100 would result if a student strongly agreed with all positively 
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stated Items and strongly disagreed with all negatively stated Items, 
A score of 20 would Indicate a set of responses which Is the exact 
converse of the set of responses for a score of 100. 

The total scale mean for the combined classes (N » 294) was 
75.29. This mean can be Interpreted as a generally favorable reaction 
across the 294 students and the 20 Items. That Is, on the averaqe for 
both students and Items, the response was favorable toward the films 
and their use. The tendency was for the students to a<iree with the 
positively stated Items and to dlsanree with the neqatlvely stated 
Items. A casual Inspection of the total scale means of the 15 
classes, however, reveals that there was fairly substantial variation 
In the degree of favorablllty. The highest mean was 92.86 and the 
lowest was 70.95, a difference of approximately two standard deviations. 
Some of this variation might be accounted for by differences In student 
characteristics. In class characteristics, or In the use of the films. 
No student characteristics could be related to the differences In class 
means because of the anonymity of student responses. However, data 
were available for considering the possible relationship of class 
characteristics and use characteristics to the degree of favorablllty 
in student reactions. These relationships are explored In the next 
two sections. 

Class characteristics and general reactions of students . The two 
class characteristics considered were class size and educational setting. 
First, as shown In Table 4, the 15 classes were placed In rank order by 
size and divided Into three groups. One group consisted of five 
classes which ranged In size from 22 to 58 students; a second group 
contained five classes which ranged In size from 11 to 20 students; a 
third group had between six and eight students within each class. The 
unweighted means for these three groups using the total scale score 
were very similar. The means for the first and second groups were 
almost Identical, 74.68 and 74.64. The mean for the third group was 
slightly higher, 79.60, but not significantly so. Class size did not 
seem to account for the variation among class means. 
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stated Items and stronqly dlsanreed with all nenatlvely stated Items. 
A score of 20 would Indicate a set of responses which Is the exact 
converse of the set of responses for a score of 100. 

The total scale mean for the combined classes (N » 294) was 
75.29. This mean can be Interpreted as a generally favorable reaction 
across the 294 students and the 20 Items. That Is, on the averaqe for 
both students and Items, the response was favorable toward the films 
and their use. The tendency was for the studen*.s to aqree with the 
positively stated items and to disanree with the neqatively stated 
items. A casual inspeci'.ion of the total scale means of the 15 
classes, however, reveal: that there was fairly substantial variation 
in the degree of favorabillty. The highest mean was 92.86 and the 
lowest was 70.95, a difference of approximately two standard deviations. 
Some of this variation might be accounted for by differences in student 
characteristics, in class characteristics, or in the use of the films. 
No student characteristics could be related to the differences in class 
means because of the anonymity of student responses. However, data 
were available for considering the possible relationship of class 
characteristics and use characteristics to the degree of favorability 
in student reactions. These relationships are explored in the next 
two sections. 

Class characteristics and general reactions of students. The two 
class characteristics considered were class size and educational setting. 
First, as shown In Table 4, the 15 classes were placed in rank order by 
size and divided into three groups. One group consisted of five 
classes which ranged in size from 22 to 58 students; a second group 
contained five classes which ranged in size from 11 to 20 students; a 
third group had between six and eight students within each class. The 
unweighted means for these three groups using the total scale score 
were very similar. The means for the first and second groups were 
almost identical, 74.68 and 74.64. The mean for the third group was 
slightly higher, 79.60, but not significantly so. Class size did not 
seem to account for the variation among class means. 
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Table 4. Average Student Rating Scale Scores by 
Size of Class Group* 



Size of Number of Unweighted 
Class Classes Grou p Mean 

22-58 5 74.68 

11-20 5 74.64 

6-8 5 79.60 



F(2. 12) « 1.433 p >.05 

The same 15 classes were divided next Into three groups according 
to their educational setting. As Indicated In Table 5, nine classes 
were at the undergraduate level, three classes were at the graduate 
level, and three grou s were In an Inservlce setting. The unweighted 
group means were remarkably similar, all three means rounded to the 
nearest whole number were 76. Certainly the educational setting did 
not explain the variation among class means. 



Table 5. Average Student Rating Scale Scores by 
E duca 1 1 onal Settina. 



Educational 


Number of 


Unweighted 


Settinq 


Classes 


Groun Mean 


Undergraduate 




76.43 


Courses 


9 


Graduate 




76.02 


Courses 


3 


Inservlce 




75.71 


Groups 


3 




F(2, 12) - 0.017 p >.05 
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within these three educational settings, however, there was less 
homogenlety of student characteristics than one inlqht expect. In the 
undernraduate classes, the vast majority of the students were nreservlce 
but within the nraduate classes the students were almost equally divided 
between preservlce teachers and active practitioners. At the Inservlce 
sites, most of the students were active practitioners. While the 
formal educational background and teaching experience of the students 
may have accounted for some of the variation amonq classes, the actual 
test of these hypotheses remains for future study because of the 
anonymity of student response Imposed for this study. 

Use characteristics and g eneral reactions of students . The 
selection of specific films, the choice of Instructional strategies, 
and the length of Instructional time devoted to the use of the films 
also varied among the 15 classes. Unfortunately, sufficient data were 
not available to test the relationship of these variables with student 
reactions except for length of 1ns):ruct1ona1 time. Three groups were 
formed based on the length of Instructional time devoted to the use of 
the films. One group consisted of five classes which devoted one hour 
to the use of the films; a second group was composed of two classes 
which used the films for two hours; a third group consisted of three 
classes which used the films for four hours. The means and analysis are 
reported In Table 6. As can be seen from this table, there was very 
little difference In reactions among the three groups. The means 
Indicated that length of Instructional time devoted to the use of the 
films had little Impact on student reactions. 



Table 6. Average Student Rating Scale Scores by Length of 
Instructional Time period. 



Group 


Instructional 


Number of 


Unweighted 


Time Period 


Classes 


Group Mean 




Hours 






1 


1 


5 


76.15 


2 


2 


2 


74.11 


3 


4 


3 


80.96 




F(2, 7) • 0.075 0 <.05 
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To separate the possible effects of total Instructional time and 
number of films used* four classes were selected each of which used 
four films but varied In the total time devoted to these four films. 
In two classes, one Instructional period was devoted to film use; In 
the other two classes, two Instructional periods were so used. A 
report of the unweighted means and associated analysis Is presented In 
Table 7. Once again, different time periods for the use of the films 
did not Indicate differences In student reactions. 



Table 7. Average Student Rating Scal e Scores by Length of 
Instructional Time period Holding Constant Number 
of Film Showings. 



Group 


Instructional 
Time Period 


Number of 
Classes 


Unweighted 
Group Mean 




Hours 






1 


1 


2 


76.30 


2 


2 

F(l, 2) ' 0.203 p 


2 

>.05 


74.n 



In sumnary, the variation In degree of favorablllty among classes 
remains to be explained. The survey did not yield enough "tryouts" to 
trace the variation among classes or uses to any specific selection of 
films, use of Instructional strategies, class characteristics, or 
student characteristics. Thus, In the remainder of this section. Item 
score means are based on combined classes and uses. That Is, since no 
differences among classes or uses were Identified In the previous 
section, classes and uses were combined for the analysis of Individual 
1 tems . 

R eactions of students on Individual scale I tems . In Table D, In 
the Appendix, Item Information based on unweighted means Is reported. 
For each Item, the minimum class mean and the maximum class mean are 
reported In addition to the unweighted class mean. A plot of this 



:70 
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Information Is portrayed In Finure 1. Once aqain, the unweighted Item 
means were rounded to the nearest whole number and assigned an adjectival 
meanlnq In a procedure similar to the handling of Information from the 
Instructor Questionnaire , The Items are listed according to their 
unweighted Item means. 

STRONGLY AGREE was not the average response for any Item. 

AGREE was the average student response for: 

"The films were worth the time we spent on them." 

"The films held my Interest." 

"The pace at which events moved In the films was about 
right." 

"The classes shown In the films appeared to be real 
cl asses r" 

"The Important concepts were clearly Illustrated In 
ine films." 

"The picture quality (color, camera focus, etc.) of 
'•he films was qood." 

"The Instructor had a thorough knowledge of the 
conctnts stressed In these films." 

"We had ample opportunity to discuss the content of 
the films In this class." 

"Generally speaking, the Individual films were arranged In 
a way that made them easy to follow." 

"The concepts emphasized In the films will be valuable to 
me In the classroom." 

"I could easily see hoi^ the films were related to one 
another. " 

"The films seemed appropriate to the general content of 
this course." 

UNCERTAIN was the average student response for; 
"I feel I learned a lot from these films." 

"The films seemed to me to be Just another bunch of 
educational films." 

6 * |_ 
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FUura I. Plot of Miniowm, Maximum and UnvalRhtad Avaraga of Claaa Meana for Kaeh JSIUfiBtlT J4UIILSi;Al£ 
Fom Ittm* 





(1) 


(2) 


(3) 


(4) 




SD 


D 


V 


A 




-1 


1 


-H 


h 



(5) 
SA 



I. The fllma vem worth tha time ve apent on thai* 
2* Tha purpose of tha filma vaa naver claar to ma* 
)• I faal I learned a lot from theat filma. 

4. tha filma held my intaraat. 

5. Theae films aaemtd to me to be Juet another bunch 
of edueetionel films* 

6. The peee at which evente moved in the filma wee 
ebout right. 

7. The cleeees ehovn in the filma appeered to be reel 
claseee. 

8. The importent concepte were cleerly illuecrated in 
the films. 

10. Ths picture quelity (color, camere focue, etc.) of 
the films was good. 

II. I would neve preferred leeming theee concepte by 
reeding or iiecueeion without the uee of filme. 

12. The concepts emphesissd in the f ilmf %«ill be valuebie 
to me in the claasroom. 

13. The content of the filme wee eemetimee confuaing. 

U. I could aeeily eee how the filme were related to 
one enother. 

IS. Oenerelly epeeking, the individuel filme were arranged 
in e wey thet made them eaay to follow. 

17. We had ample opportunity to discuss ths contwtt of ths 
films in this clsss. 

18. Ths instructor had s thorough knowledge of the concepte 
etreesed in thess films. 

20. Ths films ssemed eppropriete to the generel content of 
thie couree. 

22. the sciunJ t;**sUty of the fitma was KOod« 

M III sitMiii w«v«, M luw sie aliMMiW lUil lints. 

M. i!enf»rally, t favor the use of ediirattdnal films In 
classroom Instruction. 
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"The sound quality of the films was good." 

DISAGREE was the averaqe student response for: 

"The purpose of the films was never clear to me." 

"I would have preferred learning these concepts by 
reading or discussion without the use of films. 

"The content of the films was sometimes confusing." 

"In some ways, these films are already out of date." 

STRONGLY DISAGREL was not the average response for any Item. 

Generally, the unweighted Item means Indicate that students tended 
to agree with the positively stated Items and disagree with the nega- 
tively stated Items. This was expected and was encouraging. Amonq 
their reactions to all Items, the students were most positive about 
the picture quality of the films (Item mean » 4.05) and most uncertain 
about the sound quality of the films (Item mean » 3.37). The 
instructors perceptions were decidedly more positive on each of these 
aspects of technics 1 quality. 

A perusal of the summary of responses Indicates that the students 
reacted favorably to all but one Item concerning the conceptual content 
of the series. Finally, they aqreed with the Instructors In their 
opinion that "The classes shown In the films appeared to be real 
classes." In other words, both Instructors and students agreed on 
the general authenticity of the classroom enlsodes portrayed In the 
Pattern films and Instanced In the Concept films. 

As a final summary of student responses, percent distribution of 
student responses are reported In Table E, In the Appendix, for each 
Item. These distributions ire based on 294 students from the 15 
classes used In the final survey. An Inspection of the percent 
distribution Indicates that from 54 to 86 percent of the students 
aqreed or strongly agreed with the positively stated Items and disagreed 
or strongly disagreed with the negatively stated Items. That Is, a 
majority of the students reacted favorably to every Item. 
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Sunmary and Implications 



!n terms of the specific goals of this evaluation, to assess 
learning outcomes and reactions of users, the results can be readily 
summarized. First, use of the protocol films In the Concepts and 
Patterns series resulted in significant gains In acquisition of the 
concepts which are basic to the series. In other words, the Instructional 
use of the films led to a gain In the ability to categorize classroom 
behavior In terms of a specified sejt of concepts. Second, the Concepts 
and Patterns series was very favorably received by the Instructors (who 
used It under "consumer conditions") and was favorably received by the 
students of these Instructors. The conceptual clarity. Instructional 
utility and technical quality of the films In the series was clearly 
established for a heterogeneous sample of undergraduate and graduate, 
preservlce and Inservlce teachers. 

Implicit In thc»se general results are also certain methodological 
suggestions for the future development of protocol material «. First, 
as a means of assessing learning outcomes from protocol materials, a 
film-based test format such as the one developed and used In this study 
seems to be a valid and practical one.^ Such an Instrument provides a 
particularly advantageous means of assessing the categorization of "real 
behavior" (In lieu of printed descriptions of behavior) In a format that 
lends Itself to objective test scoring. In addition, the method of 
establishing test reliability and validity developed during this study 
(which will be reported In a test manual) should provide an Interesting 
model for future evaluators of protocol materials. 

In terms of the reactions of users. It Is most notable that from 
the viewpoint of the Instructors surveyed, the films In the Concepts and 
Patterns series were applicable to a wide variety of Instructional 



See Berliner, D. C. & Cordorl, C. A. Learning concepts about group 

f rocess ; An evaluation of protocol materials . San Franci sco » Calif. : 
ar west L^oratory for fBucatlonal Researcn and Development, 1973, 
p. 43 for reference to the development and use of a videotaped test In 
another protocol materials project. 
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methods, groups and settings. This fact, considered In light of the highly 
favorable reactions of the Instructors, Indicates that the film series Is 
highly flexible and that this flexibility Is favorably received. Further- 
more, the films fulfilled the necessary conceptual purposes of the series 
efficiently and effectively. This suggests that necessary conceptual 
content can be established and communicated In carefully designed protocol 
films with a minimal use of adjunct printed materials. The evidence 
provided In this report that the film series met these two design criteria 
(flexibility In use and Inclusion of necessary conceptual content) Is 
significant In light of the fact that the use of multi-media "packages" 
In classroom Instruction Is often such a difficult matter loglstlcally. 
Developers of protocol materials In the future might well consider the 
production of highly flexible, single medium materials. 

Obviously, a number of questions about learning outcomes and 
reactions of users remain to be explored. In the area of learning 
outcomes. It should be noted that the results of this evaluation do not 
establish that protocol fllm-based Instruction results In greater gains 
than do other forms of Instruction. Since this was not a comparative 
study of Instructional methods, the question of comparative effectiveness 
remains for further Investigation. Also remaining to be explored are 
questions about the effect on concept acquisition of using fewer or more 
of the films In the series, different film presentation sequences, etc. 
Finally, the effect of Individual variables on concept acquisition 
through protocol -based Instruction remains a question. Thp contribution 
of Intellectual, attltudlnal and motivational factors to learning out- 
comes should certainly be explored. 

In terms of the reactions of students, perhaps the most Interesting 
question left unanswered by this evaluation was the source of the 
variance In reactions of different groups to the films. As Indicated In 
the results, although student reactions were generally favorable to the 
film series, there was considerable variation In reaction among the classes 
surveyed. The contribution to this variance of the particular selection 
of students or composition of these classes, for example, remains to be 
explored. 
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REPRODUCTIVE QUESTION - a teacher question Intended to directly elicit the 
recall of content specifically learned as part of a course or 
topic of study. In response to such a question, the student 
Is expected to accurately reproduce such content or to recognize 
when It Is accurately reproduced by someone else. Typical 
student responses are repetition, restatement or recognition 
of content. 

PRODUCTIVE QUESTION - a teacher question that Is Intended to encourage the 
production of Ideas or combinations of Ideas as opposed to 
simply the reproduction of specifically learned content. A 
student response to such a question may reflect the recall of 
specifically learned content but that content Is used In such 
processes as Interpretation, application, and evaluation. 

PROBING - a teacher reaction In the form of a question or Implied question 
that pursues some aspect of the substantive content of a 
preceding student response. Such proves typically seek 
further description, clarification, explanation, or extension 
of that substantive content. By "preceding response" Is meant 
any preceding response Including, but not limited to, the 
student response that Innedlately precedes a teacher reaction. 
By "substantive content" Is meant the formal content of class- 
room discussion as opposed to such procedural content as 
assignment making, the order of discussion, or disciplinary 
matters. 

INFORMING - a teacher reaction In which Information Is Introduced that Is 

related to some aspect of the substantive content of a preceding 
student response. Such a reaction Is often Intended to produce 
some modification In the substantive content of that student 
response. By "preceding response" Is meant any preceding 
response Including, but not limited to, the student response 
that Immediately precedes a teacher reaction. By "substantive 
content" Is meant the formal content of classroom discussion 
as opposed to such procedural content as assignment making, 
the order of discussion, or disciplinary matters. 

APPROVING - a verbal and/or nonverbal teacher reaction that Is Intended to 
encourage, or might reasonably be expected to encourage 
continued student responding or a continuation of student 
behavior. 

DISAPPROVING - a verbal and/or nonverbal teacher reaction that Is Intended 
to discourage, or might reasonably be expected to discourage 
continued student responding or a continuation of student 
behavior. 
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Table A. Item-Total Correlation Coefficients and Scale Characteristics 
for the Student Rating Scale (N » 106). 





Item 

Number 


Item- Total 

Correlation 


Item 
Number 


Itein«Tota1 
Correlation 


1 


.59* 


13 


.54 


2 


.51 


14 


.71 


3 


.6b 


15 


.60 


4 


.74 


16 


omitted 


5 


.56 


17 


.49 


6 


.47 


18 


.67 


7 


.40 


19 


omitted 


8 


.57 


20 


.72 


9 


omi tted 


21 


omitted 


10 


.56 


22 


.47 


11 


.57 


23 


.38 


12 


.57 


24 


.64 



Revised Scale Characteristics: 

Number of Items » 20 

Scale Mean « 72.37 

Scale Standard Deviation » 13.65 

Scale Reliability > .89 



*A11 correlations are significant p < .05 



f 



33 



§ 



|>» 



00 



o 



CO 



«o o 



CO 



1^ 

in as 



01 

cr 



9 1 vo 



I"© 



CO 



I 



MA 



r- in 



CM 



e 



i 



t/i 

e • 

u c 
e o 

01 



• 

CO 



lO 



h 
II 



E 



O 01 
O lA 



3 O 

e m 



5 



2 
e K 



o 

S2! 



e 

9 o< 



e 4^ • 

»^ 4i 01 

^ S ^ 

tfc§ 




5 

e 

3 



o u 
e 
0) a 

8 (A 

es 
«| 

(EC 



(A 0)^ 



OIV 01 

g 13 O 

(A e u 

lA 

^ -a (A 
u 

01 « 10 



cT JO 

"So 2 



(A E 01 <A 01 y» 

f» A| S» Vi 



(A _ 

e 4^ "O 

0 a 01 

01 e ^ 



ERIC 



no 



35 



Table C. Descriptive Information from the Student Rating Scale for 
15 classes. 
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Class 
LCucatlonul 
Setting 


Number of 
Students 




Scale 




Mean 


Standard 
Deviation 


Reliability 


1 !■> jiii tmjTi tma .ri 

unaergraa 


7 






ni 
• 91 


unaergroa 






in 

IU»06 


OA 


inservi C6 


o 
o 




Q QA 

o«oO 


• 87 








A 71 


• 0«l 




43 


7Q Cfi 


Q ni 


• 83 


Gradudte 


D 




1A £Q 


Oil 

• 94 


1 Itt.jlii.ini1i ta 11 rl 

unoGrgraa 


1 1 


7ft 'Jft 


7 Ad 


•o3 


inservice 




7C; 1^ 


A A9 


AO 
• Oil 


unoergroa 


17 


71 


11 RA 


A7 

• Of 


Graduate 


20 


73.00 


9.34 


.78 


Inservice 


6 


72.00 


6.61 


.69 


Undergrad 


13 


72.00 


9.74 


.87 


Undergrad 


58 


71.86 


11.52 


.89 


Undergrad 


28 


71.43 


12.13 


.89 


Undergrad 


22 


70.95 


9.24 


.80 


Combined Classes 


294 


75.29 


11.04 


.87 



Table D. Hlnlmum, Maximum, and Unweighted Average of Class Means for 
Each Student Rating Scale Item. 
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Item 
Number 


Minimum 
Mean 


Maximum 
Mean 


Unweighted 
Mean 


1 


3.12 


5.00 


3.81 


2 


1.37 


2.36 


1 Oil 

1 .84 


3 


2.50 


4.86 


3.42 


4 


2.92 


4.86 


3.67 


5 


1 .14 


9 AO 

3.08 


A CO 

2.53 


6 


2.67 


4.37 


3.73 


7 


2.80 


f* AA 

5.00 


9 OO 

3.83 


8 


3.57 


4.43 


9 OA 

3.89 


10 


3.33 


it n*^ 

4.83 


M AC 

4.05 


11 


1.00 


2.33 


1 Al 

1 .91 


12 


3.40 


4.57 


4 Of 

3.87 


13 


1.8J 


3.36 


A il A 

2.40 


14 


3.00 


M AA 

4.29 


9 oo 

3.88 


IS 


3.50 


4.43 


4 on 

3.80 


17 


3.24 


4.37 


3.82 


18 


3.33 


4.71 


4.01 


20 


3.35 


5.00 


3.95 


22 


1.83 


4.57 


3.37 


23 


1.42 


3.09 


2.37 


24 


3.50 


5.00 


4.16 
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Instructof questionnaire bjsT COW AVAIUBU 



1 . Name 



2. School, Colleoe or Univ. 

3. Department 

4 . Address 



5. Phone f I 6. Date Form Completed 

7. In total, how many trainees (students or teachers) did you use 
the films with? 

8. Check (^) If the films were used In a colleqe or university 
settinq or In an In-service settinq: 

Colleqe or University 

In-service 



9. If used In a colleqe or university settinq, list the title (not 
the course number) of the course or courses In which the film 
or films were used and circle whether underqraduate or qraduate 
level : 

underqrad qrad 

^ undcrnrad orad 

underqrad qrad 

underqrad qrad 



10. The students In the above course(s) were (check one): 
preservlce 

experienced or active In teachinq 

both 

11. The films were used In the followlnq ways (check all that apply): 
In lecture or discussion with total class 

In Individual study outside of class 

In work with small qroups 

other (describe briefly) . 
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12. Check (>/) which films you used and Indicate the number of times 
you used eacFTllm In the spaces provided: 

Film Used Mo. of Times Used Concepts In Teacher-Punl l Inter action: 

Prob i ng & Informlncf 

Questioning ; Reproductive & Product ive 
Approving & Disapproving 
Patterns Iji Teacher-Pupil Interaction ; 
Bacteria 

People Who Help Us 
The Five Senses 
Population Control 



Essentials for Life 
Ecology 



13. Which one of the above films did you find most useful? 



14. Aporoxlmately how much Instructional time was devoted to the use 
of the films? (check one) 

one class period or one three class periods or 

hour of study or discussion three hours of study or 



discussion 

tv/o class periods or two 

hours of study or discussion four or more class perl- 

' ods or four or more hours 

of study or discussion 

PLEASE NOTE : Beginning with the next Item, the Concepts In T eacher- 
Fupil Interaction films are Identified as C oncept nims; tne Patterns 
Tn reacher-Pupll Interaction films are Identified as Pattern films. 
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15. Which of the sequences below did you qenerall.y follow In usinn 
the films? (check one) 

Concept fllin(s) follov^ed by Patter n fllin(s) 

Pattern f 11in(s) followed by Concept f11m(s) 



Conc ept f 11in(s) only 
Pattern fllin(s) only 
Other (describe) 



Please Indicate your agreement or disagreement with each of the following 
statements by checking the appronrlate space at the rlqht. 

Strongly Disagree Undecided Agree Strongly 
Disagree Aaree 

# 

16. The purpose of the Pat- 
tern films was clear. ^ 

17. The sound quality of 

the films was good. 

18. The concepts stressed 
In the Concept films 
were not of sufficient 
Importance to warrant 
the time expenditure 

Involved. 

19. The films seemed to 
hold the attention 

of my students. 

20. The olcture quality 
(color* camera focus, 
etc.) of the films 

was good. ^ 

?1. The Concept films were 
at ah appropriate level 
of difficulty for my 

students. 

22. The purpose of the 
Concept films was 

uncVeaT. ^ — 

23. The classroom episodes 

portrayed In the films i 
were not realistic or 

believable. 
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BEST COPY AVAILABLE Strongly Disagree Undecided Agree Strongly 

Disagree Agree 

24, The concepts stro^jed 
In the Concept films 
were clearly defined 
and understandable. 



25, Either the Pattern films 
or the Concept films by 
themselves could have 
achieved the same pur- 
poses. 



26. The sequence or format 
of the Concept films 
was confuslnq. 



27. Any Instructor who wants 
his students to acquire 
these concepts should 
use both the Concept films 
and Pattern films. _ 

2H. If they were available. 
I would plan to use 
these films In future 
courses. 



29. These films were obvious- 
ly produced by profession- 
al film makers. 



30. The content of the manual 
contained little relevant 
Information. 



31. The manual contained (check one): 
too little Information 



the right amount of Information 
too much Information 



32. What useful Information do you feel could be added to the printed 
manual? 
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Student SstM scsle ^^^^^ 

Instructions: Below Is a list of statements about the films you have just 
used In this class. We now w.}nt you to express your opinions about the 
films by telling us how much you agree or disagree with each of the state- 
ments below. 



Hkll 



If you strongly disagree , blacken In the space under "a' 
If you disagree but not strongly , blacken In the space under "b' 
If you are unceriaTn or nave a neutral feeling about the statement. 

blacken ln the space under "c". 
If you agree with the statement but not strongly , blacken In the 

space under "3^ 
If you strongly agree , blacken In the space under "e". 



Please Identify the name of your school, college or university and the 
name of your Instructor on the answer sheet, but do not give your name. 
Anonymity allows you to be completely candid, and it is your honesty that 
we need. 

ME WOULD LIKE TO KNOW YOUR OPINIONS ON THE FOLLOWING QUESTIONS: PLEASE 
BEGIN WITH NUMBER 1 ON YOUR ANSWER SHEET. 

1. The films were worth the time we spent on them. 

2. The purpose of the films was never clear to me. 

3. I feel I learned a lot from these films. 

4. The films held my Interest. ^ ^ , xii 

5. These films seemed to me to be Just another bunch of educational films. 

6. Thfj pace at which events moved In the films was about right. 

7. The classes shown In the films appeared to be real classes. 

8. The Important concepts were clearly Illustrated In the films. 

9. The content of the films was easy to the point of being obvious. 

10. The picture quality (color, camera focus, etc.) of the films was good. 

11. I would have preferred learning these concepts by reading or discussion 
without the use of films. , ^, ^ 

12. The concepts emphasized In the films will be valuable to me In the 
classroom. 

13. The content of the films was sometimes confusing. 

14. I could easily see how the films were related to one another. 

15. Generally speaking, the Individual films were arranged In a way that 
made them easy to follow. ^ ,^ 

16. Much of the content of these films has been an unnecessary duplication 
of other material presented In this course. 

17. We had ample opportunity to discuss the content of the films in this 
class. 

18. The Instructor had a thorough knowledge of the concepts stressed in 
these films. _ ^ 

19. We needed more opportunities to apply the content of these films. 

20. The films seemed appropriate to the general content of this course. 

21. These films were apparently produced by amateurs rather than by 
experienced film producers. 

22. The sound quality of the films was good. 

23. In some ways, these films are already out of date. ^ ^ 

24. Generally. 1 favor the use of educational films In classroom Instruction. 



